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1  Executive  Summary  of  Research  and  Findings 

The  report  summarizes  the  achievements  and  accomplishments  of  the  project  MAPLE  (Multi- 
Agent  Planning,  Learning,  and  Execution),  sponsored  by  DARPA  under  contract  P30602-98- 
2-0137.  Overall,  the  work  carried  out  under  this  contract  has  led  to  36  publications  and  a 
software  prototype  connected  to  the  Control  of  Agent  Based  Systems  (CoABS)  Grid. 

The  fundamental  problem  addressed  in  this  project  pertained  to  the  coordination  of  multi¬ 
agent  system  acting  in  dynamic,  physical  environments.  An  example  of  such  a  setting  is  a 
set  of  unmanned  ground  vehicles,  which  cooperatively  engage  in  humanitarian  missions  in 
the  aftermath  of  a  natural  disaster  (e.g.,  a  hurricane).  The  MAPLE  project  focused  on  the  de¬ 
velopment  and  evaluation  of  new  algorithms  for  integrating  information  and  for  coordinating 
the  actions  of  large  multi-agent  teams. 

Our  scientific  effort  focused  on  three  different  aspects  of  multi-agent  systems,  which  are 
outlined  in  the  detailed  approach  section  of  this  report.  Eirst,  under  the  initial  project  name 
“BORG,”  we  developed  a  prototype  of  a  secure  decentralized  database,  which  made  it  possi¬ 
ble  to  disperse  information  among  multiple  agents  in  a  secure  and  scalable  way.  Second,  we 
developed  a  multi-agent  system  for  acquiring  and  integrating  information  in  large  dynamic 
environments.  Third,  we  developed  planning  and  multi-agent  coordination  techniques  for 
enabling  large  teams  of  autonomous  agents  to  perform  missions  in  highly  dynamic  and  par¬ 
tially  observable  environments.  While  our  initial  research  focused  on  the  development  of  the 
BORG  system,  the  majority  of  resources  were  spent  developing  algorithms  for  the  last  two 
items:  techniques  for  multi-agent  information  integration,  and  techniques  for  coordinating 
many  physical  agents. 

The  resulting  algorithms  were  implemented  and  evaluated  predominately  in  the  context 
of  the  Mixed  Initiative  Agent  Team  Administration  (MIATA)  Technology  Integration 
Experiment  (TIE),  which  was  carried  out  jointly  by  a  number  of  CoABS  contractors.  We 
developed  the  backbone  software  system  for  this  TIE,  a  simulator  called  “MapleSim,”  which 
simulated  the  country  of  Honduras  after  the  devastation  caused  by  Hurricane  Mitch  in  1998. 
MapleSim  has  been  connected  to  the  CoABS  Grid,  and  a  number  of  contractors  controlled 
air  and  ground  operations  through  the  infrastructure  provided  by  the  Grid  and  MapleSim.  A 
second  contribution  to  this  demonstration  was  the  control  of  trucks  on  the  ground.  This  prob¬ 
lem  was  challenging  in  that  the  conditions  on  the  ground  were  largely  unknown;  furthermore, 
the  environment  was  highly  dynamic. 

Other  evaluations  were  carried  out  in  the  context  of  multi-robot  systems.  We  have  ex¬ 
panded  this  research  to  embrace  the  coordination  of  multiple  ground  vehicles  when  explor¬ 
ing  unknown  physical  environments,  and  the  coordination  of  multiple  unmanned  air  vehicles 
when  acquiring  information  in  dynamic  battlefield  situations  (e.g.,  finding  SAM  sites).  The 
results  of  this  project  have  successfully  been  transitioned  into  related  DARPA  projects,  such 
as  the  DARPA  TMR  project  and  the  DARPA  MICA  project. 
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2  Main  Accomplishments 

The  main  accomplishments  of  the  MAPLE  project  are  as  follows: 

•  Acc-1:  We  developed  a  prototype  of  a  secure  scalable  database,  known  as  BORG. 
Through  this  preliminary  prototype,  we  have  demonstrated  scalability  and  security  for 
this  decentralized  database. 

•  Acc-2:  We  developed  scalable  algorithms  for  multi-agent  planning  and  coordination 
in  dynamic  and  partially  observable  environments.  In  particular,  we  developed: 

-  Acc-2a:  Partially  observable  Markov  decision  process  techniques  for  coordinat¬ 
ing  many  agents  in  the  process  of  information  gathering  [15,  25]. 

-  Acc-2b:  Scalable  greedy  algorithms  for  coordinating  multi-agent  systems  when 
exploring  unknown  environments  and  acquiring  detailed  models  thereof  [9,  27]. 

-  Acc-2c:  Auction  algorithms  for  optimal  coordination  of  multiple  agents  operat¬ 
ing  in  adversarial  environments  [6] . 

-  Acc-2d:  Motion  planning  algorithms  for  deconflicting  large  number  of  moving 
entities  (e.g.,  ground  vehicles)  that  share  the  same  environment,  each  pursuing 
different  goals  [4,  5] . 

•  Acc-3:  We  developed  scalable  multi- agent  information  integration  algorithms. 

-  Acc-3a:  We  developed  a  number  of  algorithms  for  exploring  and  mapping  un¬ 
known  environments  with  teams  of  mobile  agents  [18,  19,  20,  16,  33,  34,  28]. 
The  paper  [28]  won  the  best  conference  paper  award  at  the  2000  ICRA  Confer¬ 
ence  (selected  as  the  sole  winner  from  over  1,100  submissions). 

-  Acc-3b:  We  developed  a  number  of  specialized  algorithms  for  exploring  and 
acquiring  models  of  partially  known  -  partially  unknown  environments,  such  as 
the  country  Honduras  after  the  devastation  caused  by  Hurricane  Mitch  [1,  10]. 
Our  algorithms  specifically  address  dynamic  environments,  in  which  aspects  may 
change  over  time  as  the  data  is  acquired  [10,  36]. 

-  Acc-3c:  We  developed  techniques  for  tracking  moving  entities  in  dynamic  en¬ 
vironments  with  teams  of  mobile  robots  [23,  22].  The  paper  [23]  won  the  best 
Student  Conference  paper  award  at  AAMAS,  the  international  multi-agent  con¬ 
ference. 

-  Acc-3d:  We  developed  a  number  of  advanced  probabilistic  tracking  techniques 
for  single-agent  systems  [30,  35]. 

•  Acc-4:  We  developed  MapleSim,  a  discrete  event  simulator  of  the  country  Honduras 
after  the  Hurricane  Mitch  devastated  large  parts  of  the  country  [15]. 
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•  Acc-5:  We  interfaced  MapleSim  and  our  multi-agent  coordination  system  to  the  CoABS 
Grid,  and  provided  full  support  to  various  DARPA  demonstrations  and  TIEs. 

•  Acc-6:  Finally,  we  developed  a  visualization  tool  for  the  CoABS  grid  which  was  used 
in  a  number  of  DARPA  demonstrations  and  experiments  throughout  the  CoABS  pro¬ 
gram. 
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3  Publications 


The  following  publications  directly  arose  out  of  the  MAPLE  project  and  were  fully  or  par¬ 
tially  funded  under  the  contact  F30602-98-2-0137.  The  DARPA  support  has  been  acknowl¬ 
edged. 

[1]  D.  Avots,  E.  Eim,  R.  Thibaux,  and  S.  Thrun.  A  probabilistic  technique  for  simultaneous 
localization  and  door  state  estimation  with  mobile  robots  in  dynamic  environments.  In 
Proceedings  of  the  Conference  on  Intelligent  Robots  and  Systems  (IROS),  Eausanne, 
Switzerland,  2002. 

[2]  M.  Bennewitz,  W.  Burgard,  and  S.  Thrun.  Constraint-based  optimization  of  priority 
schemes  for  decoupled  path  planning  techniques.  In  Proceedings  of  the  Joint  German 
Austrian  Conference  on  Artificial  Intelligence.  Springer  Verlag,  2001. 

[3]  M.  Bennewitz,  W.  Burgard,  and  S.  Thrun.  Optimizing  priority  schemes  for  decoupled 
path  planning  techniques.  In  Proceedings  of  the  Conference  on  Intelligent  Robots  and 
Systems  (IROS ’2001),  Hawaii,  2001. 

[4]  M.  Bennewitz,  W.  Burgard,  and  S.  Thrun.  Optimizing  schedules  for  prioritized  path 
planning  of  multi-robot  systems.  In  IEEE  International  Conference  on  Robotics  and 
Automation  (ICRA),  Seoul,  Korea,  2001.  ICRA. 

[5]  M.  Bennewitz,  W.  Burgard,  and  S.  Thrun.  Adapting  navigation  strategies  using  motions 
patterns  of  people.  In  Proceedings  of  the  IEEE  International  Conference  on  Robotics 
and  Automation  (ICRA),  2003. 

[6]  C.  Bererton,  G.  Gordon,  and  S.  Thrun.  Auction  mechanism  design  for  multi-robot 
coordination.  Submitted  for  publication,  2003. 

[7]  M.  Bema,  B.  Eisien,  B.  Sellner,  G.  Gordon,  E.  Pfenning,  and  S.  Thrun.  A  learning 
algorithm  for  localizing  people  based  on  wireless  signal  strength  that  uses  labeled  and 
unlabeled  data.  In  Proceedings  of  the  Sixteenth  International  Joint  Conference  on  Ar¬ 
tificial  Intelligence  (IJCAI),  Acapulco,  Mexico,  2003.  IJCAI. 

[8]  R.  Biswas,  B.  Eimketkai,  S.  Sanner,  and  S.  Thrun.  Towards  object  mapping  in  dynamic 
environments  with  mobile  robots.  In  Proceedings  of  the  Conference  on  Intelligent 
Robots  and  Systems  (IROS),  Eausanne,  Switzerland,  2002. 

[9]  W.  Burgard,  D.  Eox,  M.  Moors,  R.  Simmons,  and  S.  Thrun.  Collaborative  multi-robot 
exploration.  In  Proceedings  of  the  IEEE  International  Conference  on  Robotics  and 
Automation  (ICRA),  San  Francisco,  CA,  2000.  IEEE. 
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[10]  W.  Burgard,  D.  Hahnel,  R.  Triebel,  and  S.  Thrun.  Mapping  with  mobile  robots  in 
dynamic  environments.  In  Proceedings  of  the  Snowbird  Workshop  ’’Machines  That 
Learn”,  Snowbird,  UT,  2003.  NIPS  Foundation.  Extended  abstract. 

[11]  D.  Hahnel,  W.  Burgard,  B.  Wegbreit,  and  S.  Thrun.  Towards  lazy  data  association 
in  SLAM.  In  Proceedings  of  the  11th  International  Symposium  of  Robotics  Research 
(ISRR’03),  Sienna,  Italy,  2003.  Springer. 

[12]  D.  Hahnel,  D.  Fox,  W.  Burgard,  and  S.  Thrun.  A  highly  efficient  FastSLAM  algorithm 
for  generating  cyclic  maps  of  large-scale  environments  from  raw  laser  range  measure¬ 
ments.  In  Proceedings  of  the  Conference  on  Intelligent  Robots  and  Systems  (IROS), 
2003. 

[13]  D.  Hahnel,  R.  Triebel,  W.  Burgard,  and  S.  Thrun.  Map  building  with  mobile  robots 
in  dynamic  environments.  In  Proceedings  of  the  IEEE  International  Conference  on 
Robotics  and  Automation  (ICRA),  2003. 

[14]  B.  Limketkai,  R.  Biswas,  and  S.  Thrun.  Learning  occupancy  grids  of  non- stationary 
objects  with  mobile  robots.  In  Proceedings  of  the  International  Symposium  on  Experi¬ 
mental  Robotics  (ISER),  Sant’ Angelo  d’Ischia,  Italy,  2002. 

[15]  L.  Lita,  J.  Schulte,  and  S.  Thrun.  A  multi-agent  system  for  agent  coordination  in  un¬ 
certain  environments  (short  paper).  In  Proceedings  of  Agents,  Montreal,  CA,  2001. 

[16]  Y.  Liu  and  S.  Thrun.  Results  for  outdoor-SLAM  using  sparse  extended  information  fil¬ 
ters.  In  Proceedings  of  the  IEEE  International  Conference  on  Robotics  and  Automation 
(ICRA),  2003. 

[17]  M.  Montemerlo  and  S.  Thrun.  Simultaneous  localization  and  mapping  with  unknown 
data  association  using  FastSLAM.  In  Proceedings  of  the  IEEE  International  Confer¬ 
ence  on  Robotics  and  Automation  (ICRA),  2003. 

[18]  M.  Montemerlo,  S.  Thrun,  D.  Roller,  and  B.  Wegbreit.  FastSLAM:  A  factored  solution 
to  the  simultaneous  localization  and  mapping  problem.  In  Proceedings  of  the  AAAI 
National  Conference  on  Artificial  Intelligence,  Edmonton,  Canada,  2002.  AAAI. 

[19]  M.  Montemerlo,  S.  Thrun,  D.  Roller,  and  B.  Wegbreit.  FastSLAM  2.0:  An  improved 
particle  filtering  algorithm  for  simultaneous  localization  and  mapping  that  provably 
converges.  In  Proceedings  of  the  Sixteenth  International  Joint  Conference  on  Artificial 
Intelligence  (IJCAl),  Acapulco,  Mexico,  2003.  IJCAI. 

[20]  E.  Nettleton,  S.  Thrun,  and  H.  Durrant- Whyte.  Decentralised  slam  with  low-bandwidth 
communication  for  teams  of  airborne  vehicles.  In  Proceedings  of  the  International 
Conference  on  Eield  and  Service  Robotics,  Lake  Yamanaka,  Japan,  2003. 
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[21]  J.  Nieto,  J.  Guivant,  E.  Nebot,  and  S.  Thrun.  Real  time  data  assoeiation  for  Fast- 
SLAM.  In  Proceedings  of  the  IEEE  International  Conference  on  Robotics  and  Au¬ 
tomation  (ICRA),  2003. 

[22]  M.  Rosencrantz,  G.  Gordon,  and  S.  Thrun.  Deeentralized  sensor  fusion  with  distributed 
partiele  filters.  In  Proceedings  of  the  Conference  on  Uncertainty  in  AI  ( UAI),  Aeapuleo, 
Mexieo,  2003. 

[23]  M.  Rosenerantz,  G.  Gordon,  and  S.  Thrun.  Loeating  moving  entities  in  dynamic  indoor 
environments  with  teams  of  mobile  robots.  In  Proceedings  of  Autonomous  Agents  and 
Multi-Agent  Systems,  Melbourne,  Australia,  2003. 

[24]  N.  Roy  and  S.  Thrun.  Motion  planning  through  policy  search.  In  Proceedings  of  the 
Conference  on  Intelligent  Robots  and  Systems  (IROS),  Lausanne,  Switzerland,  2002. 

[25]  J.  Schulte  and  S.  Thrun.  A  heuristic  search  algorithm  for  acting  optimally  in  markov 
decision  processes  with  deterministic  hidden  state.  Internal  Report,  2001. 

[26]  R.  Simmons,  D.  Apfelbaum,  W.  Burgard,  M.  Fox,  D.  an  Moors,  S.  Thrun,  and 
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4  Detailed  Project  Report 

4.1  BORG:  Secure  Scalable  Database 

In  the  initial  12  Project  months,  the  MAPLE  project  focused  on  the  development  of  a  secure 
and  scalable  database  system.  The  idea  of  the  BORG  prototype  is  as  follows:  Imagine  you 
want  to  secure  a  piece  of  information  against  a  possible  enemy.  Then  cut  this  piece  of  in¬ 
formation  into  pieces  and  duplicate  these  pieces  among  multiple  agents.  The  key  insight  of 
the  BORG  prototype  is  that  this  dispersion  can  be  done  such  that  an  opponent  has  to  com¬ 
promise  at  least  M  out  of  N  agents  to  recover  even  a  single  bit  of  the  protected  information, 
for  arbitrary  M  <  N.  Compromising  M  —  1  is  not  sufficient;  in  fact,  no  information  can  be 
recovered  whatsoever  by  compromising  fewer  than  M  agents  (not  even  partial  information). 
By  making  M  a  large  number,  the  system  becomes  virtually  impenetrable.  It  becomes  highly 
secure  against  information  warfare  attacks. 

In  the  initial  twelve  project  months,  we  developed  a  prototype  of  this  system.  Issues 
addressed  during  this  period  pertained  to  the  manipulation  of  information  (e.g.,  adding  two 
values,  executing  simple  programs)  without  jeopardizing  the  information  itself.  The  work 
on  BORG  was  discontinued  based  on  feedback  by  the  PM  (Jim  Hendler),  who  changed  the 
focus  of  the  MAPLE  project  to  real-world  planning  and  multi-agent  coordination,  and  the 
CoABS  Grid. 

4.2  Multi-Agent  Planning  And  Coordination 

The  core  of  the  MAPLE  project  focused  on  the  development  of  multi-agent  planning  and  co¬ 
ordination  techniques,  for  coordinating  large  numbers  of  physical  agents  operating  in  highly 
unpredictable  and  uncertain  environments.  The  fact  that  those  agents  act  in  physical  spaces 
leads  to  constraints  in  which  the  physical  whereabouts  play  a  major  role  in  agent  coordi¬ 
nation.  Eurther,  the  fact  that  the  environment  is  partially  unknown  and  dynamic  mandates 
representations  that  can  account  for  the  inherent  uncertainty  in  such  environment.  As  a  re¬ 
sult,  the  agents  must  incorporate  knowledge  of  their  own  ignorance  into  decision  making, 
and  not  just  base  the  decision  on  their  best  guess  of  the  current  state  of  the  world  (as  is  com¬ 
mon  practice).  Our  approach,  thus,  differs  fundamentally  from  previous  work  on  multi-agent 
coordination,  which  has  inadequately  addressed  the  issue  of  uncertainty  and  lack  of  complete 
world  knowledge.  All  our  algorithms  are  built  on  this  insight,  and  all  of  them  address  the 
inherent  uncertainty  in  the  word. 

4.2.1  POMDP  Techniques  For  Information  Gathering  With  Many  Agents 

A  first  set  of  algorithms  described  in  [15,  25]  relies  on  a  basic  insight  known  as  partially 
observable  Markov  decision  process,  or  POMDP.  POMDPs  are  best  explained  as  follows: 
Since  the  world  is  stochastic,  it  is  best  modeled  as  Markov  process,  which  is  a  stochastic 
chain  where  the  state  of  the  world  at  time  t  depends  stochastically  on  its  state  at  time  t  —  1. 
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However,  not  all  of  the  state  is  observable.  Instead  the  agents  have  sensors  that  provide 
them  with  ineomplete  and  noisy  projeetions  of  the  (unobservable)  state.  Markov  ehains  with 
partially  observable  state  spaees  are  known  as  partially  observable  Markov  ehains.  Lastly, 
the  agents  have  to  make  eontrol  deeisions  (e.g.,  where  to  move),  henee  the  term  “deeision 
proeess”  in  POMDPs. 

Our  first  set  of  techniques,  which  formed  the  core  of  our  approach  in  the  MIATA  TIE,  is 
based  on  the  assumption  that  the  unknown  state  is  discrete:  For  example,  roads  in  Honduras 
might  be  passable  or  not  after  Hurricane  Mitch  destroyed  much  of  the  county’s  infrastructure; 
bridges  might  have  collapsed.  The  challenge  then  is  to  reach  a  number  of  goal  locations 
(e.g.,  population  centers)  under  this  initial  uncertainty  about  the  situation  in  the  POMDPs. 
In  [25],  we  developed  efficient  search  algorithms  for  finding  optimal  exploration-exploitation 
plans,  which  not  only  provided  a  single  best  route  to  a  set  of  goals,  but  also  a  full  set  of 
contingencies  on  what  to  do  should  a  path  be  blocked.  On  top  of  this,  we  developed  a  multi¬ 
agent  arbitration  scheme  for  assigning  individual  agents  to  individual  goals.  This  algorithm 
evaluated  the  contingency  plans  provided  by  the  individual  agents,  and  assigned  agents  to 
goals  based  on  the  expected  time  of  arrival  (in  fact,  based  on  the  entire  probabilistic  profile 
as  to  when  an  agent  could  be  assumed  to  arrive  at  a  goal).  In  [15],  we  increased  the  efficiency 
of  this  approach  by  several  orders  of  magnitude  through  the  introduction  of  abstraction, 
in  which  entire  subparts  of  a  large  POMDP  were  solved  first  and  represented  by  a  single 
complex  arc  in  an  abstract  model  of  the  world.  All  of  these  approach  were  brought  to  bear 
in  the  MIATA  TIE,  and  they  were  thoroughly  evaluated  [25]. 

4.2.2  Coordinating  Multi-Agent  Exploration 

In  [9,  27],  we  further  developed  multi-agent  exploration  techniques  which  we  subsequently 
implemented  on  physical  robots.  The  following  image  shows  one  of  our  robot  systems  (Fig¬ 
ure  1): 


Figure  1:  Robots  (left)  and  map  with  exploration  path  (right). 

The  problem  addressed  in  this  research  is  a  restricted  version  of  the  POMDP  above.  In 
particular,  the  environment  is  static  and  the  robot  can  greedily  explore  it.  However,  what 
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makes  this  case  difficult  is  the  extremely  large  number  of  unknowns:  The  environment  is 
represented  by  a  grid  consisting  of  tens  of  thousands  of  grid  cells.  Here  the  agents  have  to 
rely  in  their  sensors  to  infer  the  complex  structure  of  the  environment  as  they  go.  The  sensor 
interpretation  and  data  fusion  has  to  take  place  in  real-time. 

We  have  developed  scalable  techniques  for  doing  all  this.  In  particular,  our  approach 
coordinates  multiple  robots  during  exploration  and  mapping.  It  interprets  sensor  data  and 
enables  the  robots  to  acquire  a  single,  combined  map.  As  described  in  detail  in  [9,  27],  our 
approach  involves  an  auction-style  algorithm  in  which  agents  “bid”  for  individual  places  to 
explore.  A  high-level  agent  then  assigns  agents  to  places.  This  assignment  is  constantly 
revised  to  accommodate  the  ever-changing  state  of  knowledge  of  the  robot  system.  Empiri¬ 
cal  comparisons  in  [9,  27]  show  that  our  approach  is  superior  to  uncoordinated  exploration 
techniques,  commonly  found  in  previous  literature. 

4.2.3  Auction  Algorithms 

Carrying  this  work  a  step  further,  we  have  recently  developed  auction-style  multi-agent  co¬ 
ordination  algorithms  that  are  more  powerful  [6].  In  particular,  in  past  work  by  us  and  others 
(including  the  work  above),  the  coordination  was  based  on  a  fixed  set  of  prices  for  shared 
commodities  (e.g.,  a  fixed  reward  for  exploring  an  unknown  location).  However,  in  cases 
where  multi-agent  systems  share  complex  commodities  (e.g.,  multiple  trucks  in  Honduras 
have  to  share  a  limited  supply  of  fuel),  the  costs  for  using  such  resources  in  auction-style 
algorithms  is  usually  not  known.  In  fact,  one  can  argue  that  setting  appropriate  action  costs 
amounts  to  solving  the  harder  part  of  the  problem.  Once  such  costs  have  been  determined, 
the  problem  is  easily  decomposed  into  single-agent  planning  and  execution  problems. 

Our  approach  in  [6]  gets  at  the  core  of  this  issue:  It  alternates  a  conventional  individual- 
agent  planning  phase  with  a  phase  in  which  an  auctioneer  determines  the  optimal  costs  to 
assign  to  individual  commodities  and  constraints  that  arise  between  multiple  agents.  In  this 
way,  a  much  broader  class  of  problems  is  made  amenable  to  multi-agent  auction-style  algo¬ 
rithms,  and  the  overall  performance  of  the  system  is  improved.  We  have  demonstrated  this 
in  the  contact  of  a  number  of  simulated  multi-agent  problems  described  in  [6]. 

4.2.4  Motion  Planning  for  Large  Numbers  of  Ground  Vehicles 

In  [4,  5],  we  have  specifically  addressed  the  problem  of  multiple  physical  agents  sharing  the 
same  physical  environment.  Here  we  model  ground  vehicles  that  have  to  deconflict  their 
motion  through  the  environments.  Such  problems  are  well-known  to  all  of  us;  Just  consider 
the  problem  of  tens  of  thousands  of  people  commuting  to  work  through  a  limited  network  of 
streets  and  intersections. 

We  have  developed  a  randomized  set  of  algorithms  that  improves  over  the  previous  state 
of  the  art.  Our  approach  uses  prioritized  planning  techniques  to  develop  an  initial  multi-agent 
plan,  which  is  subsequently  refined  by  improving  the  order  at  which  individual  vehicles 
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can  determine  their  loeal  routes.  The  approaeh  was  tested  in  a  eomplex  network  of  roads 
and  interseetions  reminiseent  (but  not  identieal)  to  the  seenario  used  in  the  MIATA  TIE. 
We  have  shown  that  our  approaeh  is  signifieantly  more  sueeessful  in  deeonflieting  vehiele 
motion  than  elassieal  multi-agent  path  planning  algorithms.  However,  our  approaeh  assumes 
a  deterministie  world,  whieh  makes  it  less  powerful  than  the  POMDP  algorithms  deseribed 
above  (in  faet,  it  really  addresses  a  different  problem).  Detailed  deseriptions  of  the  results 
ean  be  found  in  [4,  5]. 

4.3  Multi-Agent  Information  Integration 

In  addition  to  multi-agent  planning  and  eoordination  algorithms,  we  developed  a  suite  of 
sealable  multi-agent  information  integration  algorithm.  The  problem  underlying  all  this  work 
is  easily  formulated:  If  N  agents  operate  in  a  dynamie  and  uneertain  world,  how  ean  they  ef- 
fieiently  integrate  their  loeal  observations  into  a  single,  eonsistent  world  knowledge.  One  ob¬ 
vious  solution  would  be  to  design  a  eentral  information  agent  (a  “master  agent”)  with  whieh 
all  agents  eommunieate.  However,  sueh  a  system  would  be  eharacterized  by  a  single  point 
of  failure;  moreover,  it  would  be  subjeet  to  sealing  limitations  due  to  the  eommunieation 
overhead  involved  for  the  master  agent.  Our  work  is  strietly  deeentralized:  Communieation 
oeeurs  only  between  neighboring  agents,  and  the  global  model  emerges  through  loeal  eom¬ 
munieation.  The  resulting  algorithms  seale  to  any  number  of  agents  and  are  highly  reliable 
against  failure  of  individual  network  eomponents.  We  view  this  work  as  an  important  step 
towards  the  vision  of  network-eentrie  eomputing.  Further,  this  work  is  a  cornerstone  of  the 
multi-agent  systems  developed  in  the  MAPLE  program. 

4.3.1  Mapping  Unknown  Environments 

In  a  number  of  publieations  [18,  19,  20,  16,  33,  34,  28],  we  have  developed  algorithms  for 
information  integration  in  the  eontext  of  map  aequisition  of  unknown  environments.  This 
problem  is  one  of  the  hardest  in  robot  pereeption:  It  involves  vehieles  that  measure  nearby 
entities  as  they  move  about  their  environments,  seeking  to  model  it  in  a  eonsistent  manner.  It 
has  an  important  limiting  assumption,  namely  that  the  environment  is  statie.  This  assumption 
is  eommon  in  the  literature  on  mapping;  below  we  will  diseuss  our  approaehes  for  dynamie 
environments. 

Possibly  the  most  important  result  of  the  MAPLE  projeet,  doeumented  in  [20,  16,  31, 
33,  34],  is  a  teehnique  for  information  integration  that  is  truly  sealable  to  any  number  of 
agents.  The  idea  is  to  represent  the  model  probabilistieally,  by  the  posterior  over  all  maps 
given  the  measured  data.  Further,  these  approaehes  represent  this  probabilistie  estimate  in 
its  information  form,  whieh  technically  amounts  to  a  log-likelihood.  Information  is  additive. 
Nearby  agents,  thus,  ean  add  their  information,  and  pass  the  sum  on  to  other  neighbors.  This 
large  network  of  information- adding  entities  ean  aeeommodate  any  number  of  agents. 

Our  work  has  speeifioally  foeused  on  keeping  the  message  sizes  small  even  if  the  poste- 
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rior  is  complex.  In  problems  where  the  sensors  move  (e.g.,  trucks  move  through  Honduras), 
the  exaet  updates  grow  over  time.  This  is  beeause  uneertainty  in  vehiele  motion  ean  affeet 
inereasingly  large  fraetions  of  the  map.  We  have  developed  approximate  algorithms  that 
eommunieate  the  gist  of  new  information  immediately,  but  delay  the  eommunieation  per¬ 
taining  to  remote  loeations.  In  doing  so,  the  message  size  is  kept  small;  in  fact,  its  size 
is  independent  of  the  size  of  the  world  and  the  total  sensor  history.  However,  one  of  the 
key  questions  is  whether  the  resulting  posterior  is  still  eonsistent  with  the  one  that  would  be 
obtained  by  eostly  eommunieation  algorithms  with  large  message  sizes.  We  have  shown — 
theoretieally  and  empirieally — that  this  is  still  the  ease.  Further,  we  have  aehieved  exeellent 
empirieal  results  when  eompared  to  the  non- approximate  (but  eostly)  solution  of  communi- 
eating  all  sensor  data  baek  to  a  master  agent.  The  papers  [34,  31]  develop  this  approaeh  for 
a  single  agent  system.  Work  in  [33,  16]  extends  this  approaeh  to  multi-agent  system,  under 
various  flavors  of  uneertain  knowledge.  The  paper  [20]  deseribes  our  effieient  eommuniea¬ 
tion  teehnique.  This  work  was  earned  out  in  eooperation  with  researehers  at  the  University 
of  Sydney,  and  sinee  has  been  implemented  for  teams  of  unmapped  air  vehieles. 

4.3.2  Modeling  Dynamic  Environments 

In  [1,  10,  36],  we  have  developed  a  number  of  algorithms  partieularly  suited  for  modeling 
dynamie  environments.  The  key  idea  here  is  that  some  of  the  environment  is  static  (e.g.,  the 
layout  of  a  place),  whereas  other  quantities  ean  ehange  over  time.  This  work  overeomes  a 
eritieal  assumption  made  in  the  information  integration  work  deseribed  thus  far,  in  that  it 
allows  for  non-statie  aspeets  of  the  environment  to  exist. 

The  work  in  [1]  applies  a  so-called  Rao-Blaekwellized  particle  filter  to  the  problem  of 
modeling  the  state  of  eertain  dynamic  objects.  It  is  cast  as  the  problem  of  learning  the  sta¬ 
tus  of  doors  in  an  indoor  environment  with  a  mobile  robot,  but  it  is  equally  applieable  to 
the  problem  of  learning  the  state  of  bridges  and  roads  in  the  MIATA  TIE.  More  relevant  is 
the  work  in  [10,  36],  whieh  addresses  the  problem  of  acquiring  maps  of  environments  with 
statie  and  dynamie  aspeets.  The  paper  [10]  applies  the  expeetation  maximization  algorithm 
(EM)  to  separate  statie  from  dynamic  aspects,  and  then  simply  applies  a  eonventional  map¬ 
ping  algorithm  for  modeling  the  static  aspects  of  the  environment.  In  [36],  we  model  both, 
statie  and  dynamic  aspects.  The  teehnique  detects  and  tracks  moving  objects  in  the  environ¬ 
ment,  and  makes  those  track  part  of  the  overall  model.  We  have  demonstrated  all  three  of 
those  algorithms  in  the  eontext  of  physieal  robot  implementations,  using  indoor  and  outdoor 
robots.  In  our  experiments,  we  found  that  models  of  unpreeedented  eomplexity  eould  be 
aequired — in  fact,  we  know  of  no  other  work  that  would  address  modeling  issues  in  dynamie 
environments  of  similar  eomplexity. 
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4.3.3  Tracking  Moving  Entities  With  Many  Agents 

In  [23,  22],  we  address  the  problem  of  traeking  moving  objeets  under  massive  oeelusion,  with 
large  numbers  of  moving  agents.  The  motivating  seenario  was  one  by  whieh  an  intruder 
seeks  to  move  through  an  environment  undeteeted.  Mobile  robots  have  to  move  through 
the  environment  in  ways  that  guarantee  that  such  an  intruder  be  found.  This  is  yet  another 
example  of  information  integration  in  a  dynamic  environment.  Static  information  integration 
techniques  are  inapplicable. 

Our  approach  relies  on  a  carefully  designed  communication  protocol  between  neighbor¬ 
ing  agents.  Agents  analyze  their  local  beliefs.  They  query  neighbors  with  hypotheses  (e.g., 
intruder  took  path  X)  that  they  deem  to  be  likely  given  their  observations.  Neighbors  then 
respond  with  confirming  or  contradicting  evidence  is  such  evidence  is  available;  otherwise 
they  communicate  the  query  to  their  own  neighbors.  The  resulting  algorithm  was  found  to 
be  highly  efficient  in  integrating  information  in  such  dynamic  environments.  Physical  tests 
with  4  robots  were  complimented  with  simulations  of  up  to  100  robots.  We  found  that  the 
communication  overhead  could  be  cut  down  by  several  orders  of  magnitude  when  compared 
to  a  centralized  system,  where  each  agent  communicates  all  observations  to  a  central  high- 
level  agent.  Further,  this  algorithm  is  a  first  scalable  algorithm  to  tracking  moving  entities 
with  vast  amounts  of  occlusion,  with  virtually  any  number  of  agents. 

4.3.4  Advanced  Probabilistic  Tracking  Techniques  For  Single-Agent  Systems 

Finally,  we  have  developed  a  number  of  algorithms  for  tracking  belief  states  in  POMDPs 
that  advance  the  state  of  the  art  in  specific  directions  [30,  35].  In  [30],  we  developed  a 
particle  filter  tracking  algorithm  that  considers  the  performance  task  at  hand:  Instead  of  just 
maximizing  the  accuracy  of  the  tracked  belief  state,  it  does  so  relative  to  the  expected  cost  of 
tracking  errors.  The  result  is  quite  different  from  traditional  tracking  algorithms:  By  taking 
the  costs  into  account,  the  computational  resources  are  mostly  focused  to  parts  of  the  space 
where  failures  and  errors  matter  the  most.  We  are  not  aware  of  previous  work  that  would 
fold  costs  into  the  basic  tracking  system. 

In  [35],  we  further  extended  this  idea  to  incorporate  hierarchical  representations.  The 
problem  addressed  here  is  tracking  in  the  context  of  complex  diagnostic  systems,  in  which  a 
system  can  have  a  huge  number  of  possible  failure  states.  Tracking  all  of  them  is  prohibitively 
expensive.  The  approach  in  [35]  relies  on  abstraction  to  track  groups  of  possible  failures,  and 
refines  those  tracks  dynamically  as  additional  information  arises.  This  approach  has  success¬ 
fully  been  applied  to  online  diagnostics  of  the  Hyperion  robot,  a  robot  recently  deployed  in 
a  desert  in  Chile. 
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4.4  MapleSim:  A  Simulator  of  Ground  and  Air  Vehicles  in  Honduras 


The  MAPLE  team  played  a  major  role  in  the  support  of  the  MIATA  TIE.  In  addition  to  the 
algorithms  deseribed  above,  we  developed  a  diserete  event  simulation  platform  known  as 
MapleSim  [15].  A  view  of  this  system  is  shown  in  Eigure  2: 


Eigure  2:  Sereenshots  of  the  MapleSim  simulation  environment  (Honduras). 

In  essenee,  this  simulator  ean  model  any  geographie  region  based  on  data  widely  avail¬ 
able  trough  the  USGS  and  other  serviees.  The  simulator  models  air  and  ground  vehieles. 

It  also  models  storms  and  their  effeet  on  the  environment’s  infrastrueture.  In  partieular,  it 
models  destruetion  of  navigation  infrastrueture  and  humanitarian  requirements  arising  out  of 
problems  sueh  as  ground  water  eontamination.  The  MapleSim  system  formed  the  baekbone 
of  the  MIATA  TIE,  in  whieh  it  was  used  to  model  the  situation  in  Honduras  after  Hurrieane 
Miteh  destroyed  up  to  90%  of  the  infrastrueture.  It  has,  however,  been  used  for  a  number  of 
other  geographie  regions,  by  a  number  of  eontraetors  in  the  DARPA  CoABS  program. 

The  development  and  support  of  MapleSim  eonsumed  about  a  third  of  the  overall  re- 
sourees  provided  under  this  eontraet.  The  simulation  platform  eonsists  of  a  eore,  whieh  ean 
be  run  under  the  Windows  or  Einux  operating  system.  The  eore  eonneets  to  a  graphieal  visu¬ 
alization  tool  ealled  “SimSpy,”  whieh  makes  it  possible  to  visualize  the  situation  and  affeet  it 
(e.g.,  by  manually  destroying  bridges  in  Honduras).  The  system  also  possesses  a  full  API  for 
eontrolling  individual  vehieles,  and  reeeiving  sensor  data  eolleeted  by  these  vehieles.  The 
types  of  vehieles  modeled  in  our  approaeh  inelude  ground  vehieles  (trueks),  fixed  wing  aireraft 
and  rotorerafts.  The  simulator  ean  model  up  to  100  sueh  vehieles  at  a  time,  at  user-seleeted 
time  seales.  In  addition,  MapleSim  offers  a  limited  model  of  natural  disaster  effeets,  and  the 
resulting  damage  to  the  road  strueture.  This  faeility  sets  MapleSim  aside  from  a  number  of 
alternative  simulation  tools,  whieh  tend  to  model  the  environment  as  more  statie  (and  there- 
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Figure  3:  Architecture  of  the  MIATA  TIE. 


fore  more  predictable).  The  MapleSim  software  has  been  made  available  via  our  project  Web 
site  http  :  /  / www .  cs  .  emu  .  edu/ ~maple. 

4.5  Porting  and  Interfacing  with  CoABS  Grid 

Major  effort  was  spent  on  porting  software  to  the  CoABS  Grid.  We  developed  a  C/C++ 
interface  that  formed  the  core  of  another  contractor’s  work  on  porting  software  to  the 
Grid.  Based  on  this  interface,  we  made  a  number  of  services  available  for  use  by  others: 
the  MapleSim  simulator,  and  our  agent-based  control  system  for  coordinating  large  number 
of  ground  vehicles.  The  system  played  a  major  role  in  the  MIATA  TIE,  in  which  multiple 
contractors  cooperatively  studied  the  use  of  the  Grid  and  agent-based  technology  in  highly 
dynamic  and  unpredictable  environments.  The  porting  efforts,  however,  were  not  fully  com¬ 
pleted  for  our  most  recent  software  developments. 

4.6  CoABS  Grid  Visualization  Tools 

Einally,  we  developed  a  graphical  visualization  tool  to  monitor  and  analyze  the  message 
flow  in  the  Grid.  This  tool  applied  a  number  of  graph  visualization  algorithms  to  visualize 
the  complex  and  dynamic  network  of  interacting  entities  in  the  CoABS  Grid.  This  tool  was 
used  in  a  number  of  TIEs  throughout  the  CoABS  Program. 
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